Associating signal intelligence to objects via residual reduction

ABSTRACT

Generally discussed herein are systems and apparatuses that are configured to and techniques for associating a SIGINT signal with an object or 
     tracklet. According to an example, a technique can include (1) estimating a first set of times, each time of the first set of times can indicate how much time it would take for a respective SIGINT signal of a set of SIGINT signals to travel from a point on a tracklet extracted from video data to a respective collector, (2) estimating a second set of times corresponding to times at which the video data corresponding to the point on the tracklet was gathered, or (3) associating the set of SIGINT signals with a tracklet of the plurality of tracklets based on the first set of times, the second set of times, and a set of ToAs of SIGINT signals at the plurality of collectors.

RELATED APPLICATION

This application claims priority to U.S. Provisional Application Ser.No. 61/788,504, filed Mar. 15, 2013, which is incorporated herein byreference in its entirety.

TECHNICAL FIELD

Examples generally relate to associating a SIGnal INTelligence (SIGINT)signal to a moving object, such as an object that can be moving or isbeing or has been tracked. More specifically, examples relate toincreasing the speed (decreasing the time) which it takes to associate aSIGINT signal or a set of SIGINT signals to an object or tracklet.

TECHNICAL BACKGROUND

SIGINT can be data gathered through signal interception or analysis.SIGINT can be broken into two broad categories: (1) COMmunicationsINTelligence (COMINT) and (2) ELectronic signal INTelligence (ELINT).COMINT deals with data gleaned from messages or voice information, whileELINT deals with data gleaned from non-communication sensors (e.g.,Global Positioning System (GPS) sensors).

BRIEF DESCRIPTION OF THE DRAWINGS

In the drawings, which are not necessarily drawn to scale, like numeralsmay describe similar components in different views. Like numerals havingdifferent letter suffixes may represent different instances of similarcomponents. The drawings illustrate generally, by way of example, butnot by way of limitation, various embodiments or examples discussed inthe present document.

FIG. 1 shows an example of a system for associating a SIGINT event witha tracklet (i.e. track fragment or at least partial track).

FIG. 2 shows an example of overlapping residual areas (e.g., confidenceintervals) for each of a plurality of objects.

FIG. 3 shows an example of a graph of distance versus distance of tenobservations of three objects travelling in the same general direction,each object travelling at a different speed.

FIG. 4 shows an example of a table that summarizes averages of residualerrors determined using ten observations of high quality SIGINT signalssent from the objects shown in FIG. 3 and also shows nine bar graphs,each depicting a distribution of calculated residual errors so as tohelp visualize how much overlap or separation occurs between the exampleresidual error calculations for the objects.

FIG. 5 shows an example of a table that summarizes averages of residualerrors determined using ten observations of medium quality SIGINTsignals sent from the objects shown in FIG. 3 and also shows nine bargraphs, each depicting a distribution of calculated residual errors soas to help visualize how much overlap or separation occurs between theexample residual error calculations for the objects.

FIG. 6 shows an example of a table that summarizes averages of residualerrors determined using ten observations of low quality SIGINT signalssent from the objects shown in FIG. 3 and also shows nine bar graphs,each depicting a distribution of calculated residual errors so as tohelp visualize how much overlap or separation occurs between the exampleresidual error calculations for the objects.

FIG. 7 shows an example of a graph of distance versus distance oftwenty-five observations of three objects travelling in the same generaldirection, each object travelling at a different speed.

FIG. 8 shows an example of a table that summarizes averages of residualerrors determined using ten observations of low quality SIGINT signalssent from the objects shown in FIG. 7 and also shows nine bar graphs,each depicting a distribution of calculated residual errors so as tohelp visualize how much overlap or separation occurs between the exampleresidual error calculations for the objects.

FIG. 9 shows an example of a technique for associating a SIGINT signalwith a tracklet.

FIG. 10 shows an example of a graph of distance versus distance of threeobjects travelling at the same general speed.

FIG. 11 shows an example of a table that summarizes averages of residualerrors determined using ten observations of perfect quality (i.e. novariance due to signal error) SIGINT signals sent from the objects shownin FIG. 10 and also shows nine bar graphs, each depicting a distributionof calculated residual errors so as to help visualize how much overlapor separation occurs between the example residual error calculations forthe objects.

FIG. 12 shows an example of a table that summarizes averages of residualerrors determined using ten observations of medium quality SIGINTsignals sent from the objects shown in FIG. 10 and also shows nine bargraphs, each depicting a distribution of calculated residual errors soas to help visualize how much overlap or separation occurs between theexample residual error calculations for the objects.

FIG. 13 shows an example of a table that summarizes averages of residualerrors determined using ten observations of low quality SIGINT signalssent from the objects shown in FIG. 10 and also shows nine bar graphs,each depicting a distribution of calculated residual errors so as tohelp visualize how much overlap or separation occurs between the exampleresidual error calculations for the objects.

FIG. 14 shows an example of a technique for associating a SIGINT signalwith an object or a tracklet.

FIG. 15 shows a block diagram of an example of a machine upon which anyof one or more techniques (e.g., methods) or processes discussed hereinmay be performed.

DESCRIPTION OF EMBODIMENTS

Examples in this disclosure relate to associating a SIGINT event with anobject on a tracklet extracted from video data. More specifically,examples can relate to apparatuses, systems, and techniques fordetermining which tracklet a SIGINT event originated from, such as in anenvironment with a high density of SIGINT transmitters or a high densityof movers or associated tracklets. Even more specifically, examples canrelate to apparatuses, systems, or techniques for speeding up a processof associating a SIGINT signal with an object or tracklet.

Exploiting SIGINT to determine a location of an emitter is a commonpractice. Stationary emitters can be integrated over time to achieveimproved accuracy and can present a less challenging version of aproblem solved using a technique disclosed herein.

Moving emitters are more challenging because, at least in part, theuncertainty in motion of the emitter. The motion of the emitter canlimit the ability to integrate over time. A situation that can be evenmore challenging can include associating a SIGINT signal to a movingemitter in an environment with a high density of moving emitters thatare emitting SIGINT signals. The challenge is due to, at least in part,the combination of large confidence regions associated with estimatedlocations of the emitters. The large confidence region can be due tomotion of the emitter and ambiguity that can be caused by multipleemitters, such as by confidence regions (e.g., residual regions orellipses) associated with a location where a SIGINT signal originatedincluding multiple emitters. The large confidence region can encompassmany emitters simultaneously, thus making the association of a SIGINTevent with a track (e.g., a tracklet extracted from video data) notdeterministic (see FIG. 2 and the corresponding description for a moredetailed explanation of overlapping confidence regions).

Another complication to associating a SIGINT signal with an object, suchas beyond multiple emitter objects existing within a residual, caninclude a tracklet being considered variable, such as within a durationof observations (e.g., video data). A sum-of-squared residuals based onthe number of observations can be scaled to help overcome this problem.Only tracks with durations that support an “adequate” number ofpotential associated SIGINT events may be considered, such as can beimplemented in getTracksfromMovers pseudocode, such as is presentedherein.

Exploiting SIGINT to support multi-hypothesis tracking (e.g.,associating a plurality of distinct sets of SIGINT signals with arespective object) has been previously unsuccessful for moving emittersin high density environments. Previous attempts to associate a SIGINTevent with a mover (e.g., a moving object or emitter) in an urbanenvironment has failed, at least in part, because of large regions ofuncertainty and a density of emitters (e.g., transmitters that emit aSIGINT signal) within the confidence region.

Advances in video tracking can improve the resolution and accuracy oftracks. Signal processing techniques can support the association ofSIGINT events based on electronic content and externals. Combining theability to associate movers with tracks and associate SIGINT events to acommon emitter (e.g., an emitter with a known location at a given time)with the concept of reduction or minimization of residuals (e.g.,reducing a difference in expected and observed Time Of Arrival (ToA) foran emitter at a known location or collectors at predictable or knownlocations over time) can help in associating a SIGINT event with anemitter.

One or more approaches to associating a SIGINT event with an object ortrack disclosed herein investigates hypotheses that associate a SIGINTevent to an emitter location over time (e.g., from a track extractedfrom video data) by computing the sum-of-square residuals for each ofthe hypotheses. Each hypothesis can be the association of one or moreSIGINT events with a specific track and the residual can indicate arelative probability that the SIGINT event actually originated from anemitter on the track.

An interpolation of tracks to times that support the observedtimes-of-arrival at a set of collectors can provide a constrainedgeo-location problem that, when solved, can help the association of aSIGINT signal to an object or tracklet. One or more approaches discussedherein differ from previous attempts that associate and exploitconfidence ellipses resulting from multiple SIGINT events over time.Previous attempts calculate a region in which a SIGINT signal isexpected to have originated from and if the confidence region includesonly one emitter or object then the SIGINT signal is assumed to haveoriginated from that emitter. This approach does not work inenvironments with emitters that are relatively close together so thatmultiple emitters are in the confidence region. Efforts to aggregate aseries of SIGINT events to reduce ambiguity have been stymied, at leastin part, due to the high density of emitters and the limited capacity toform tracks in high density environments.

Solutions to a SIGINT to mover association problem that include trackingin a cluttered environment can include the exploitation of as much apriori information as possible on a sensor as well as on the targetsand, in particular, on the terrain to enhance track quality and trackcontinuity.

Although one or more cases may remain challenging, such as cases inwhich a variance of the residuals of the correct SIGINT to moverassociation is large and the distance between objects is small, or casesthat include a small number of SIGINT events and relatively agile objectmotion, this disclosure identifies scenarios where exploitingintermediate geo-location results can provide a mechanism to achieveSIGINT to target track data or object association (e.g., in an optimalsense).

Reference will now be made to the FIGS. to help further explain detailsregarding how to associate a SIGINT event with a tracklet, such as in anenvironment that can include multiple moving emitters that can be withinthe same confidence region or overlapping confidence regions.

FIG. 1 shows an example of a system 100 that can include a plurality ofobjects 102A, 102B, and 102C, each of the objects 102A-102C can includean emitter configured to transmit a SIGINT signal 104A, 104B, and 104C,respectively. The objects 102A-C can each be moving or stationary. Eachof the objects 102A-102C can be filmed (singularly or jointly) andtracklets 106A, 106B, and 106C can be extracted from the video datacollected from the filming. The system 100 can include a plurality ofcollectors 108. The collectors 108 can have a known location at a giventime (e.g., the location can be known to within a predictable accuracyif the exact location of the collector is not known). The location ofthe collectors 108 can commonly be called a collector ephemeris. Thecollectors 108 can each be stationary or moving.

Consider a case where the following assumptions hold: 1) a number ofSIGINT events over time, are from the same emitter; and 2) there aretrack fragments over the same period of time, that determine (e.g.,unambiguously) the location of movers at specific times within thetargeted time interval. As used herein, a SIGINT event is a SIGINTsignal 104A-104C and a plurality of arrival times of the SIGINT signal104A-104C, one arrival time at each of the plurality of collectors 108.

Given a collection of sets of SIGINT events from one-to-many collectors108 (e.g., with an associated ephemeris), the quality of individualSIGINT events (optional), and a collection of tracklets 106A-106C (e.g.,track fragments) for moving objects 102A-102C, a maximum likelihood canbe applied to determine an assignment (e.g., a best assignment) ofSIGINT events (e.g., SIGINT signals 104A-C) to track fragments, such asthe tracklets 106A-C. The residuals of the trial associations, anintermediate result to conventional geo-location, can provide anindicator of the confidence of each assignment. As a result, theseresiduals can be used to both select a set (e.g., an optimal set) ofassociations, or can provide additional information, such as a physicalfeature measurement of an object 102A-C, to support multi-hypothesistracking.

A position of the object 102A along each tracklet 106A-C can beinterpolated or predicted, such as by modeling how long it would takefor the SIGINT signal 104A to travel from a transmitter associated withthe object 102A-C to the respective collector 108. The ToA, such asshown at 110A and 110B, of the SIGINT signal 104A (e.g., t1 and t4 forSIGINT signal 104A) can be used to interpolate where the object 102A-Cwould have been on each tracklet 106A-C at the time the SIGINT signal104A was transmitted. A modeled time it would take for the SIGINT signal104A to travel from the transmitter to the collector 108 can becalculated. This modeled time can be used to estimate a point on atracklet 106A-C that the emitter was at the time it transmitted theSIGINT signal 104A. The point on the tracklet 106A-C that is modeled canbe the point that reduces (e.g., minimizes) a residual error (e.g., sumof squared differences) of expected ToAs and actual ToAs across all thecollectors 108. The modeled time can subtracted from the ToA to get atime of transmission with a residual (e.g., an ellipse or other shapethat defines a confidence interval that the transmission occurred atthat time). The respective time of transmission or location on thetracklet 106A-C can be compared to determine the residual error. Thetracklet 106A-C corresponding to the lowest residual error can beassumed to be the tracklet 106A-C that the SIGINT signal 104A originatedfrom.

For an individual SIGINT event, a collection of observedtime-of-arrivals (TOAs), such as at 110A-B, can be generated, one foreach of a plurality of collectors 108. For a given location in space, aphysical model can predict the expected ToA at each collector 108, suchas by modelling how long it would take a SIGINT signal 104A-C to arriveat the collector 108. The time corresponding to the modeled time can beused, such as along with a time corresponding to when the video data wasgathered, and compared to actual ToAs of the SIGINT signal 104A-C ateach of the plurality of collectors 108. The sum of a squared difference(e.g., a residual) between the physical model and the actual time can bean estimate of the likelihood of that location (e.g., tracklet 106A-Cassociated with the location) being associated with the SIGINT signal104A-C (relative to alternatives).

In summary, a geo-location technique can use maximum likelihood toiterate to a local minimum over these squared residuals. Thegeo-location technique can combine SIGINT events for stationaryemitters, or moving emitters. Motion determined by tracks, or thetracklets 106A-C, can be combined with the associated SIGINT events, toachieve association of SIGINT to movers, such as can be characterized bytrack observations through video over time.

Given an ephemeris (e.g., location of collectors 108), and an ability tointerpolate ephemeris to any time in a specified time window, can allowthe collector 108 locations for a collection of ToAs to be predicted.Note that collectors 108 can be mobile and need not be stationary.

Given the ability to identify tracklets 106A-C within a time or locationwindow, and the ability to interpolate tracklets 106A-C based on acollection of ToAs for a SIGINT event, the sum-of-squared residualsrelated to a track and a collection of SIGINT events can be computed.

In some embodiments, it can be beneficial to interpolate where an object102A-C was located on a specific tracklet 106A-C at a specific time,such as a time that occurred between video frames of video data thatcaptured the track. Interpolating tracks may not be as simple as linearinterpolation between observables. There may be no “on the ground” timefor a SIGINT event without knowing the location of the emitter (e.g.,object 102A-C in or on which the emitter resides or is transmittingfrom). Interpolating a location on a tracklet 106A-C based on a set ofSIGINT observables (e.g., ToAs) can correspond to finding the locationalong the tracklet 106A-C (e.g., a constrained portion of the tracklet106A-C that remains after a bounding region that constrains the regionin which the SIGINT event could have originated has been determined),which can reduce (e.g., minimize) residuals over all possible locations.This can be viewed as a constrained geo-location where locations arelimited to points along the tracklet 106A-C.

Time can remain a factor, as no assumptions about constant motion, orother motion limitations have been made. In one or more examplesdiscussed herein, it can be assumed that tracklets 106A-C are sampled ata high enough resolution that ambiguity in tracklet 106A-C route is nota factor. In one or more examples discussed herein, it can be assumedthat variations in object 102A-C speed do not greatly impact theinterpolation process. In the process pseudo-code presented below,interpolateTrack can be configured to perform a track interpolationfunction if a track interpolation is needed or desired.

FIG. 2 shows an example of the objects 102A-C and their associatedresiduals 212A, 212B, and 212C, respectively. When a prior SIGINT eventto tracklet 106A-C association process is applied to the SIGINT eventsfrom objects 102A-C, the process could clearly associate SIGINT eventsfrom the residual 212A with a transmitter associated with the object102A. However, the objects 102B and 102C can be more troublesome. As canbe seen, the residuals 212B and 212C both contain the objects 102B and102C within them, such as near the center of the residual 212B and 212C(e.g., confidence intervals). Prior SIGINT association processes wouldnot be able to determine which object 102B or 102C a SIGINT signal 104Bor 104C originated from in the example shown in FIG. 2. One or more ofthe processes described herein can, sometimes very accurately, determinewhich object 102B or 102C (or tracklet 106B or 106C associated with theobject 102B or 102C, respectively) the SIGINT signal 104B or 104Coriginated from.

A description of pseudo-code configured for associating a SIGINT eventwith an object 102A-C of a plurality of objects, such as can helpdetermine which SIGINT signal 104B-C is associated with which object102B-C as shown in FIG. 2, is presented as follows:

Program Title: AssociateSIGINT

Inputs: A set of N SIGINT observations with locations: L={μ₁, . . . ,μ_(N)} and covariances Σ={Σ₁, . . . , Σ_(N)}; A set of N sets of TOAmeasurements, TOA^(j): For each SIGINT observation j, a set of TOAmeasurements: TOA^(j)={TOA_(i) ^(j)}, 1≦i≦|TOA^(j)|; A set of N SIGINTobservation times: T={t₁, . . . , t_(N)}; Collector ephemeris E; orSubroutine getTracksfromMovers that identifies tracklets that occur in atime and location window.Output: An object index and the computed residuals for that object. Ifthe SIGINT observations can be determined to be associated with astationary object, M=−1 and the residuals correspond to a fixedlocation.Program Description: Given a set of SIGINT events from a singletransmitter, along with the corresponding ephemeris and ToA observablesfrom the collectors, the ability to identify and extract tracklets forthe time window and location window containing the SIGINT events,exploit the sum-of-squared residuals to find the most likely trackassociated with the SIGINT event. Also, compare the alternativeassumption of a stationary emitter against the best mover track.

An example of pseudo code configured to implement the foregoingdescription is presented as follows:

Program Start: R ← getBoundingBox((μ₁, Σ₁), ... , (μ_(N), Σ_(N)))%generate box containing SIGINT events C ← getTracksfromMovers(R, [t₁,t_(N)]) %identify tracklets that occur in the box and time window For i= 1:N  temp1 = 0;  For j = 1:|C|   Location = interpolateTrack(C_(j),t_(i))   %interpolate track j location at current event time t_(i)   $\left. {{temp}\; 1(j)}\leftarrow{{{temp}\; 1(j)}\; + \frac{{getResidualError}\left( {{Location},{TOA}^{j},E,t_{i}} \right)}{{TOA}^{j}}} \right.$ END FOR END FOR moverIndex ← arg min{temp1} % index of most likelymover MoverResidual ← min({temp1}) %sum of residuals for most likelymover stationaryResidual ← getMinResidual(R, TOA, E, T) %get residualsfor best stationary location IF MoverResidual > stationaryResidual moverIndex ← −1  Residual = stationaryResidual ELSE  Residual =MoverResidual END IF Return (moverIndex, Residual)

The processing in the AssociateSIGINT function can identify one or morelikely objects 102A-C or tracklets 106A-C, such as in a bounding box(e.g., bounding area or region) determined by one or more of the SIGINTevents, such as in the collection of SIGINT events.

Alternate assumptions can include that the events are associated with noemitter or object 102A-C in the location window (e.g., bounding box), orthe events are associated with a stationary emitter. The formerassumption can be based on the likelihood of multiple events all fallingoutside of the confidence regions for each individual event. Thisassumption can be discarded. The latter assumption can be retained. Thiscan be addressed by finding a likely location (e.g., the most likelylocation) of a stationary emitter within the bounding box (e.g.,location window), such as based on a sum-of-squared residuals for SIGINTevent observables. This can be accomplished using conventionalgeo-location, or by a grid search, among other techniques. The residualsfor this likely stationary location can then be compared against anobject 102A-C (e.g., the best object, such as can be associated with thesmallest residual error).

AssociateSIGINT can return a mover index (e.g., or a constant, such as“−1” or another constant, or other indicator if it is determined thatthe object 102A-C is stationary) and a corresponding sum-of-squareresiduals. AssociateSIGINT can return the sum-of-square residuals forobjects 102A-C or tracklets 106A-C, the likely or best stationarylocation, or corresponding residuals. This information can supportidentifying potential ambiguity between objects 102A-C (e.g., withresiduals that are close in magnitude or overlap). The results for thestationary location can provide confidence information for thehypothesis “stationary or not stationary”. In one or more embodiments,the residuals can be normally distributed, resulting in the sum ofsquared residuals being distributed Chi-square with N−2 degrees offreedom for N observations.

The following description and example pseudo code describes an exampleof a description of a pseudo code configured to calculate a residual foreach object 102A-C or tracklet 106A-C being associated with a givenSIGINT event:

Subroutine Title: getResidualError

Inputs: A location l at time t; A set of N TOA measurements: {TOA₁ . . ., TOA_(N)}, one for each collector 108; or Ephemeris E.

Output: An error sum S

Description: Given TOA observations for N collectors, and a location land time t, compute the sum-of-squared residuals that originate from theassumption that a signal was emitted at time t and location l, resultingin the collection of TOAs at the collectors.

An example of pseudo code configured to implement the foregoingdescription is presented as follows:

Subroutine Start:

c ← getCollectorLocations(E, t)  %Note: there will be N collectorlocations S ← 0 FOR i = 1:N   S ← S + |getExpectedTOA(c(i),l) −TOA_(i)|² END FOR RETURN S

The following description and example pseudo code describes an exampleof code configured to calculate a minimum residual from all of thecalculated residuals:

Subroutine Title: getMinResidual

Inputs: A region, R; a set of sets of TOA measurements, {TOA₁₁, TOA₁₂, .. . TOA_(ln(l))}, . . . , {TOA_(k1), TOA_(k2), . . . TOA_(kn(k))};ephemeris E; and a set of times T={t₁, . . . , t_(N)}

Output: An error sum S

Description: Find the location, x, in region R that minimizes thesum-of-squared TOA residuals across all collectors and all SIGINT events(e.g., across all TOA observations associated with the same SIGINTsignal 104A-C). This can be achieved using conventional geo-locationmaximum likelihood search, or as a brute force grid search.

An example of pseudo code configured to implement the foregoingdescription is presented as follows:

Subroutine Start:

$S = {\min\limits_{x \in R}\left\lbrack {\sum\limits_{i}\;{{getResidualError}\left( {x,{TOA}_{i,:},E,t_{i}} \right)}} \right\rbrack}$RETURN S

AssociateSIGINT can apply a weight (e.g., an equal weight, differentweight, or a combination thereof) to observables (e.g., SIGINT signals104A-C or events). Covariance matrices can be used to weight theobservables, such as can be based on an inverted covariance (e.g., poormeasurements can be weighted less than better measurements). Suchweighting can result in improved performance when some observables areof poorer quality than others.

getExpectedTOA represents a physical model that predicts the expectedobservables based on emitter and collection locations and can includeenvironmental factors. This physical model can incorporate the effect ofterrain into its estimates, as elevation can directly impact thepredicted ToA. Terrain can be used to determine visibility between thecollector 108 and emitter (e.g., object 102A-C), resulting in “large”residuals for cases with little or no visibility. This can be an issuefor collectors 108 at lower elevations.

Test cases were analyzed and simulated to demonstrate the efficacy ofthe processes or techniques and associated systems or apparatusesconfigured to associate SIGINT to objects 102A-C, as discussed herein.The test cases presented herein include three objects starting at thesame location, moving in the same general direction, and each object ismoving at a different speed (as shown in FIG. 3). These cases weredemonstrated for ten associated SIGINT events (FIG. 3) and twenty-fiveassociated SIGINT events (FIG. 7). The twenty-five associated SIGINTevents help demonstrate a benefit of more observables (e.g., SIGNTevents). The priority application, U.S. Provisional Application Ser. No.61/788,504, discusses a few other scenarios, namely, (1) three objectsstart at the same location and move in different directions (180 degreesand 90 degrees from each other) at the same speed; and (2) three objectsstart at different locations and move in the same relative direction atthe same speed. Results for these two scenarios are also shown anddiscussed in the priority application, which is incorporated herein byreference in its entirety.

FIG. 3 shows an example of a graph 300 of distance in miles vs distancein miles of three objects (e.g., object 1, object 2, and object 3 asshown in the example of FIG. 3). The three objects move in the samegeneral direction with each object having a different speed.

FIG. 4, FIG. 5, and FIG. 6 each show a table, TABLE 1, TABLE 2, andTABLE 3, respectively, that summarizes the average residual error of thenine bar graphs (“1-1”; “1-2”; “1-3”; “2-1”; “2-2”; “2-3”; “3-1”; “3-2”;and “3-3”) on each of the respective FIGS. 4, 5, and 6. Graphs labelled1-1 show the residual calculated in a situation where the correct object102A-C (e.g., or tracklet 106A-C associated with the object 102A-C) isobject one (e.g., the tracklet 106A-C associated with object one) (the“1” before the dash) and that the SIGINT events from object one arebeing used to determine the residuals (the “1” after the dash). Graphslabelled 1-2 show the residual calculated in a situation where thecorrect object 102A-C (e.g., or tracklet 106A-C associated with theobject 102A-C) is object one (e.g., the tracklet 106A-C associated withobject one) and that the SIGINT events from object two are being used todetermine the residuals (the “2” after the dash). All the bar graphs arelabeled using the same labeling convention. Thus, graphs labelled “3-2”show the residual calculated in a situation where the correct object102A-C (e.g., or tracklet 106A-C associated with the object 102A-C) isobject three (e.g., the tracklet 106A-C associated with object three)and that the SIGINT events from object two are being used to determinethe residuals (the “2” after the dash).

FIG. 4 shows residuals calculated using the objects as depicted in FIG.3 using high quality SIGINT signals (e.g., SIGINT signals 104A-C thathave relatively high Signal to Noise Ratio (SNR), no noise, or highmagnitude at the collector 108); FIG. 5 shows residuals calculated usingthe objects as depicted in FIG. 3 using medium quality SIGINT signals104A-C that have a lower SNR or lower magnitude than the high qualitySIGINT signals 104A-C of FIG. 4; and FIG. 6 shows residuals calculatedusing the objects as depicted in FIG. 3 using low quality SIGINT signals104A-C that have a lower SNR or lower magnitude than the medium SIGINTsignals 104A-C of FIG. 5. This series of FIGS. 4, 5, and 6 shows thatthe processes discussed herein can associate SIGINT events or signalswith objects 102A-C or their associated tracklets 106A-C, such asobjects 102A-C that are relatively close together (e.g., within lessthan about five hundred feet of each other), accurately. The separationof objects that can be tolerated by the process can depend on number orlocation of collectors 108, object 102A-C movement or object 102Amovement relative to another object 102B, number of SIGINT eventsreceived from a particular emitter, or combinations thereof. In general,the greater the variety in movement, the greater the number ofcollectors 108, the greater the number of SIGINT observations, thecloser the objects 102A-C can be and still be distinguished from oneanother. FIGS. 4, 5, and 6 also demonstrate that as the SIGINT signalquality is reduced, the accuracy of the process is also reduced. This isdue, at least in part, to the confidence interval of an object 102A-Clocation being reduced (and the resulting residual ellipse gettingbigger) as the quality of the SIGINT signal 104A-C received is reduced.

FIG. 7 shows an example of a graph 700 of distance in miles versusdistance in miles substantially similar to that shown in FIG. 3, withthe graph in FIG. 7 showing twenty-five observation locations plottedfor each object rather than ten. FIG. 8 shows residuals calculated usingthe objects as depicted in FIG. 7 with low quality (e.g., high variance)SIGINT signals 104A-C that have a higher SNR or lower magnitude than themedium SIGINT signals 104A-C, such as the SIGINT signals used to producethe graphs in FIG. 5. As can be seen by comparing FIG. 8 to FIG. 6, byusing more observables (e.g., SIGINT events) better separation betweenresidual distributions can be achieved. A difference between FIGS. 6 and8 is the number of observables used in calculating the residuals.Separation between the distributions in the bar graphs of FIG. 8 isgenerally greater than the separation between the distributions in thebar graphs of FIG. 6. This greater separation indicates that the numberof errors in associating a tracklet with a set of SIGINT events can bereduced by increasing the number of observables. Thus, increasing thenumber of observables used to associate the tracklet with a set ofSIGINT events can increase the accuracy and reduce the number of errorsin associating SIGINT signals 104A-C with objects 102A-C or tracklets106A-C.

For each graph in FIGS. 4-6 and 8 a thousand observable samples weregenerated and observables were corrupted by random error of controlledvariance (low, medium, and high corresponding to the signal quality, thehigh variance was for low signal quality, the medium variance was formedium signal quality and the low variance was for high signal quality).The average of the residuals for each candidate track versus the correcttrack are presented in TABLE 1 of FIG. 4, TABLE 2 of FIG. 5, TABLE 3 ofFIG. 6, and TABLE 4 of FIG. 8. The distribution of observed residualsfor each track combination is presented so that the amount of overlapbetween distributions can be observed (in graphs “1-1”; “1-2”; “1-3”;“2-1”; “2-2”; “2-3”; “3-1”; “3-2”; and “3-3” of each of the FIGS. 4-6and 8). For example, results for 1-1 can correspond to the observedresiduals when track 1 is the correct track, and track 1 is selected.This can be non-zero due, at least in part, to the errors added toreflect a physical model inaccuracy. The physical model can be of aSIGINT signal 104A-C travel medium.

FIG. 9 shows an example of a technique 900 for associating a SIGINTsignal 104A-C with an object 102A-C or tracklet 106A-C. At 902, ToAs ateach of a plurality of collectors 108 of a first signal from each of aplurality of moving transmitters can be estimated. Each first signal canbe transmitted from a transmitter on a tracklet 106A-C extracted fromvideo data and received at the plurality of collectors 108. The locationof each of the plurality of collectors 108 can be known, predetermined,or discernible. Estimating the ToAs at the plurality of collectors 108can include (1) estimating a first time, the first time indicating howlong it would take the first signal to travel from a point on thetracklet 106A-C to a collector 108, (2) determining a second time, thesecond time indicating the time at which the transmitter was at thepoint on the tracklet 106A-C; or (3) determining an estimated ToA at thecollector 108 as a function of the first time and the second time.

At 904, each estimated ToA can be compared to a respective actual ToA ofa SIGINT signal 104A-C received at each of the collectors 108. At 906, alikelihood that the first signal corresponds to the SIGINT signal 104A-Ccan be determined, such as to determine whether the SIGINT signal 104A-Cwas transmitted from a transmitter (e.g., emitter) on the correspondingtracklet 106A-C.

The technique 900 can include generating a bounding area. Tracklets106A-C that eventually have related residual errors calculated can bewithin the bounding area, such as within a specified time window, suchas to constrain the number of tracklets 106A-C to calculate residualsbased on or constrain tracklets 106A-C to those within the boundingarea. The bounding area can include a geographical region in which aSIGINT event is estimated to have originated from as a function of anestimated location or a corresponding covariance defining a confidenceinterval that the estimated location is the actual location the SIGINTevent originated from. Each of the plurality of SIGINT events caninclude a SIGINT signal 104A-C and the actual ToA of the SIGINT signal.The tracklets 106A-C can be determined to each be active in the boundingarea in a time window. The time window can be determined as a functionof the actual ToAs of SIGINT signals 104A-C at the plurality ofcollectors 108.

The technique can include determining a plurality of residual errors,one residual error for each tracklet 106A-C of a plurality of trackletsper SIGINT event. Each residual error can represent a likelihood that aSIGINT event originated from a respective object 102A-C on the tracklet106A-C. The residual error can be determined as a function of (1) aninterpolated location of the transmitter at a specified time, (2) theactual ToAs at each collector 108 of the SIGINT event, and (3) thelocation of each collector 108. The interpolated location can bedetermined based on tracklet 106A-C data from full motion video.

The technique 900 can include determining the SIGINT signal 104A-Coriginated from an object 102A-C on the tracklet 106A-C that correspondsto a lowest residual error of the plurality of residual errors. Thetechnique 900 can include determining if it is more likely that theSIGINT signal 104A-C originated from a moving transmitter or astationary transmitter.

Techniques, apparatuses, or systems discussed in this disclosure canreduce the processing time of associating a SIGINT event or signal104A-C with an object 102A-C as compared to a prior technique,apparatus, or system. The time between receipt of observables (e.g.,SIGINT events or signals 104A-C) across multiple intelligence sources tothe association of the observable with an object 102A-C can be reduced.Accelerating this process can expand the application space of thedisclosure, possibly to the point where tracks or tracklets 106A-C orthere associations with SIGINT signals 104A-C can be improved to nearreal-time associations.

The SIGINT association can iterate through trial locations on the ground(i.e. on the tracklet 106A-C) to reduce a residual error betweenexpected Time of Arrival (ToA) at the collector 108, such as based on aphysical model or an actual ToA at the collector 108. The tracklets106A-C considered can be limited to a candidate or constrained tracklet106A-C set within a bounding area, such as a bounding area determinedbased on an estimated location of where the relevant SIGINT signal104A-C originated from or a covariance that defines a confidence levelof the estimated location being the actual location of origin of theSIGINT signal 104A-C.

Consider a case where associated SIGINT observations (e.g., computedToAs at collectors 108 or Time Differences of Arrival (TDoAs) of SIGINTsignals 104A-C at collectors 108 determined to be from the same emitter)are the last arriving variables to a SIGINT association process. Giventhe collector 108 positions, such as collector 108 positions over time,and tracklet 106A-C information (e.g., data defining the tracklets106A-C from video, such as full motion video), the processing time ittakes from the arrival of the ToAs and the association of the SIGINTsignals 104A-C to tracklets 106A-C can be reduced. This can beaccomplished, such as by computing or transforming (e.g., pre-computingor pre-transforming) the track observation time from the video to apredicted observation time at the collectors 108. This computation canhelp simplify a later association of a process of associating the SIGINTsignals 104A-C to the tracklets 106A-C. These transformed observationscan be used to generate estimated residuals using the transformed orcomputed SIGINT signal 104A-C times at the collectors 108, allowing forresidual optimization within a limited number of computations (from thearrival time of the associated SIGINT observations, such as the SIGINTToAs). Residual reduction applied to the transformed or computedcollector 108 times can be equivalent to, or as beneficial as, residualreduction or optimization as applied to locations on the tracklets106A-C.

A physical model can be used to estimate ToAs at the collector 108. Thiscan be followed by computations of residuals. In conventionalgeo-location, a gradient (i.e. a change in a residual based on a changein location on the ground) can be used to update to a new location ofthe object 102A-C or tracklet 106A-C. Instead of repeatedly estimatingToAs at the collector 108 based on a physical model, residuals can becomputed at a new time offset at the collectors 108 after the ToAs atthe collectors 108 are received. This may include interpolation betweenobservations if the sampling resolution (e.g., of the video or theSIGINT reception at the collector 108) is inadequate.

A latency of associating a SIGINT signal 104A-C to an object 102A-C ortracklet 106A-C can be reduced by converting tracklet 106A-C observationtimes (e.g., the time the video data corresponding to the tracklet106A-C was recorded) to an expected or estimated collector 108observation time (e.g., the time the SIGINT signal 104A-C is received atthe collector 108, such as prior to receiving the associated SIGINTobservations.

For an individual SIGINT event, a collection of associated observed ToAsof the SIGINT signal 104A-C associated with the SIGINT event at thecollector 108 can be generated, such as one for each collector. This canbe represented as follows:

S_(i)={TOA_(i1), TOA_(i2) . . . TOA_(ij) . . . TOA_(iN)}, where i is aSIGINT signal 104A-C index and j is a collector 108 index and there areN collectors 108.

For each candidate tracklet 106A-C, a set of locations over time canform the track, which can be represented as follows:

T_(m)={(X_(m1), Y_(m1), Z_(m1), T_(m1)), . . . , (X_(mKm), Y_(mKm),Z_(mKm), T_(mKm))} where m is a tracklet 106A-C index with K_(m),observations with associated location (X_(m), Y_(m), Z_(m)) at timeT_(m).

Given an ephemeris of the collectors 108, or an ability to interpolateephemeris to a time, such as in a specified time window, the collector108 locations at a specified time can be (estimated) as follows:

C_(n)={X_(n), Y_(n), Z_(n), T_(n)} where n is a collector 108 index withcollector 108 location (X_(n), Y_(n), Z_(n)) at time T_(n).

For any given location in space, and a specified transmission time, aphysical model can help predict or estimate an expected ToA of theSIGINT signal 104A-C S_(i) at each collector 108. The transmission timesand locations of a transmitter that transmits the SIGINT signal can beunknown. A set of observed objects 102A-C, that can include a potentialemitter, and their locations at a subset of times can be known.Transforming the tracklet 106A-C observations (e.g., location or time)to predicted collector 108 observations can be represented as follows:

T_(m)→{t_(mn1), . . . , t_(mnKm)} where m is the tracklet 106A-C index,n is the collector 108 index, and there are K_(m) observations of thetracklet 106A-C m.

The tracklet 106A-C observation times and SIGINT signal 104A-Cobservation times may not match perfectly. In such cases, the tracklet106A-C observations can be up-sampled, such as by assuming relativelyconsistent motion between tracklet 106A-C observations. Such up-samplingcan help get a tracklet 106A-C observation closer in time than anon-up-sampled set of tracklet 106A-C data. Other methods ofinterpolation between tracklet 106A-C observations or SIGINTobservations can be used to provide estimates closer in time than wouldotherwise be available, such as to provide an estimate adequately closein time.

The sum of the squared differences (i.e., the residual) between thephysical model time and the actual observation times can provide anestimate of the likelihood of that tracklet 106A-C (e.g., location)being associated with the SIGINT event, such as relative toalternatives. Due to errors in the observations, perfect alignment oftracklet 106A-C times transformed to collector times with actual ToAobservations at collectors 108 may not be attained.

Assume that for each SIGINT event K there are up to M SIGINTobservations {S_(1K), S_(2K), . . . , S_(MK)}, where M is the number ofcollectors 108 with observations for this SIGINT event. A tracklet106A-C time offset (Δ) can be determined, for each tracklet 106A-C i,that minimizes the sum of squared residuals as follows:

${RES}_{i,k} = {\min\limits_{\Delta}\left( {\sum\limits_{j = 1}^{M}\;\left( {{TOA}_{jk} - t_{{ij}\;\Delta}} \right)^{2}} \right)}$

Summing these residuals across a plurality of SIGINT events associatedwith the same tracklet 106A-C or object 102A-C can provide an estimateof the “fit” of the tracklet 106A-C to the associated SIGINT events. Acomplication to this process can include tracklets 106A-C being variablein the duration of observations. To account for this, the sum-of-squaredresiduals can be scaled based on the number of observations. Onlytracklets with durations that support an “adequate” number (e.g.,threshold number, such as a predetermined number) of potentialassociated SIGINT events (e.g., N, where N can vary for each SIGINTevent) can be considered as follows:

${TOTAL}_{{RES}_{i}} = \frac{\sum\limits_{k}\;{RES}_{i,k}}{N}$

Processing can be further accelerated by filtering out tracklets 106A-Cthat are likely not associated with a SIGINT event or SIGINT signal104A-C. If the SIGINT event has a corresponding computed locationestimate or ground time (e.g., estimated transmission time), (S_(Xk),S_(Yk), S_(Tk)), then tracklets 106A-C can be excluded by removingtracklets with a distance from the estimated observation location thatis above a threshold. A derivation of a distance parameter to compare tothe threshold can be as follows:

Assume a bivariate Gaussian distribution with parameters (μ_(x), μ_(y),σ_(x), σ_(y), ρ) are derived from the threshold percentage (e.g., apercentage between 90 percent to 100 percent, such as 95 percent, or alower percentage, such as 50 percent, 75 percent, 80 percent, or otherpercentage) confidence ellipse parameters (x, y, a, b, θ):

μ_(x) = x μ_(y) = y$\sigma_{x} = \sqrt{\left( {\cos\;{\theta\left( \frac{a}{2.4477} \right)}} \right)^{2} + \left( {\sin\;{\theta\left( \frac{b}{2.4477} \right)}} \right)^{2}}$$\sigma_{y} = \sqrt{\left( {\sin\;{\theta\left( \frac{a}{2.4477} \right)}} \right)^{2} + \left( {\cos\;{\theta\left( \frac{b}{2.4477} \right)}} \right)^{2}}$$\rho = \frac{\sin\;\theta\;\cos\;{\theta\left( {\left( \frac{b}{2.4477} \right)^{2} - \left( \frac{a}{2.4477} \right)^{2}} \right)}}{\sigma_{x}\sigma_{y}}$

A BiVariate Normal (BVN) score to compare to the threshold can bedefined as:

${{BVN}\left( {i,j} \right)} = {\frac{1}{2{\pi\sigma}_{x}\sigma_{y}\sqrt{1 - \rho^{2}}}{\exp\left\lbrack {- \frac{z}{2\left( {1 - \rho^{2}} \right)}} \right\rbrack}\mspace{14mu}{where}}$$z = {\frac{\left( {x_{i} - \mu_{x}} \right)^{2}}{\sigma_{x}^{2}} - \frac{2{\rho\left( {x_{i} - \mu_{x}} \right)}\left( {Y_{i} - \mu_{y}} \right)}{\sigma_{x}\sigma_{y}} + {\frac{\left( {Y_{i} - \mu_{y}} \right)^{2}}{\sigma_{y}^{2}}.}}$

The BVN score can be viewed as the probability that the tracklet 106A-Cobservation matches the SIGINT observation. BVN scores for multipleassociated SIGINT events (e.g., SIGINT events known to originate fromthe same object 102A-C or transmitter) can be obtained by multiplyingthe individual scores. Low scoring tracklets 106A-C (e.g., tracklets106A-C that have a BVN score lower than a predefined threshold) can beexcluded from further processing. By adjusting the threshold higher morepossible tracklets 106A-C can be removed from processing and processingtime can be decreased. The threshold may not be set too high such thatno tracklets 106A-C attain a score higher than the threshold.Alternatively, the threshold can be defined such that a score less thanthe threshold means that the tracklet 106A-C is sufficiently close tothe location to be considered in a process of associating a SIGINTsignal 104A-C with the tracklet 106A-C

A previously described approach performed residual reduction on eachSIGINT event in an associated set of SIGINT events to find a likelytracklet 106A-C to associate with each SIGINT event. The residualreduction for each SIGINT event can be done using a transformed time atthe collector 108 and a linear search. Note that for unconstrainedgeo-location a linear time search may not be adequate as the search isacross multiple dimensions (e.g., three if altitude is included).Because tracklets 106A-C can limit the search to one dimension at anyinstant in time (i.e., one track observation to the next trackobservation) then constrained search on the ground can be substantiallyequivalent to a linear search at the collectors 108 (e.g., in collector108 time).

With pre-computed time observations at the collector, the process canfunction faster, such as with no additional information than waspreviously used.

In some examples, such as examples with high video frame ratesassociated with video data of the tracklets 106A-C, it can be assumedthat the tracklet 106A-C observations are errorless. In these examples,spacings between tracklet 106A-C observations can be equally errorless.

In one or more examples, the tracklet 106A-C observations can be “slid”along with the SIGINT signal 104A-C observations. This process can besimplified by generating equally spaced tracklet 106A-C observations,such as can be similar to or present in full motion video. Sliding atime offset (e.g., by a sample or fraction of a sample) on the ground(e.g., on a tracklet 106A-C) can be equivalent to sliding time in thetransformed (expected) collector 108 observation times. A constant timeoffset on the ground can mean a variable offset at the collectors 108.This variability can be accounted for in the transformation process, orinterpolation between the transformed points.

FIG. 10 shows a line graph of distance versus distance that depictsmotion of three objects (object 1, object 2, and object 3). Simulationswere performed to demonstrate the benefit of using residuals toassociate SIGINT to movers can be preserved when working with tracktimes transformed to an expected arrival time at a collector 108 foreach collector 108.

FIGS. 11, 12, and 13 show nine bar graphs (nine bar graphs (“1-1”;“1-2”; “1-3”; “2-1”; “2-2”; “2-3”; “3-1”; “3-2”; and “3-3”), where eachgraph is labeled using the same convention as discussed with regard toFIGS. 4, 5, and 6.

FIG. 11 shows residuals calculated using the objects as depicted in FIG.10 using high quality SIGINT signals (e.g., SIGINT signals 104A-C thathave relatively high Signal to Noise Ratio (SNR), no noise, or highmagnitude at the collector 108); FIG. 12 shows residuals calculatedusing the objects as depicted in FIG. 10 using medium quality SIGINTsignals 104A-C that have a lower SNR or lower magnitude than the highquality SIGINT signals 104A-C of FIG. 11; and FIG. 13 shows residualscalculated using the objects as depicted in FIG. 10 using low qualitySIGINT signals 104A-C that have a lower SNR or lower magnitude than themedium quality SIGINT signals 104A-C of FIG. 12. This series of FIGS.11, 12, and 13 shows that the processes discussed herein can associateSIGINT events or signals 104A-C with objects 102A-C or tracklets 106A-C,such as objects that are relatively close together (e.g., within lessthan about five hundred feet of each other), accurately. These FIGS. 11,12, and 13 also demonstrate that as the SIGINT signal 104A-C quality isreduced, the accuracy of the process is also reduced. This is due, atleast in part, to the confidence interval of an object 102A-C locationbeing reduced (and the resulting residual ellipse getting bigger) as thequality of the SIGINT signal 104A-C received is reduced. The bar graphsshow the distribution over one thousand trials. Overlaps between bargraphs labeled with the same beginning number (e.g., “1-1”, “1-2”, and“1-2”) indicate regions where possible errors in associating a SIGINTevent with a tracklet 106A-C or object 102A-C can occur. The residualsin the bar graphs of FIGS. 11, 12, and 13 were computed by convertingtracklet 106A-C times or locations to collector 108 times or distancesfrom the tracklet 106A-C observations to the collector 108, thenperforming residual reduction by effectively sliding (i.e. evaluatingthe effect of varying ground time offsets on residuals) against thetransformed observations in time. A brief description of sliding againsttransformed observation in time is presented. In summary, a computationof a residual is performed by summing a squared difference betweencollector 108 observation times and a predicted arrival time (at eachcollector) based on a location on the tracklet 106A-C n the ground. Thearrival times (ToAs) of the SIGINT signals 104A-C can be predicted inadvance. If a video image time on the ground matches the predicted timeof arrival is easily estimated. If the time on the ground is betweenpredicted observations then an interpolation may be done. If a linearinterpolation at the collectors is assumed to be adequate, such as canbe based on the frequency of video samples, then interpolating a timeoffset of a certain percentage on the ground can result in nearly thesame percentage of time offset can be observed at the collectors 108. Asliding scale simply moves the collector time in this linear (andsometimes in non-linear) fashion based on the time offset on the ground.Sliding refers to applying this “variable ruler” at the collectors toachieve the same effect as evaluating the change in location on theground (for each SIGINT event separately).

FIG. 14 shows a technique 1400 for associating a SIGINT signal 104A-Cwith an object 102A-C or tracklet 106A-C. At 1402, a first set of timescan be estimated. Each time of the first set of times can indicate howmuch time it would take for a respective SIGINT signal 104A-C of a setof a plurality of SIGINT signals to travel from a point on a tracklet106A-C of a plurality of tracklets extracted from video data to arespective collector 108 of a plurality of collectors at determinablelocations, such as within a known or calculable accuracy. At 1404, asecond set of times corresponding to times at which the video datacorresponding to the point on the tracklet 106A-C was gathered can beestimated. At 1406, the set of SIGINT signals can be associated with atracklet 106A-C of the plurality of tracklets based on the first set oftimes, the second set of times, and a set of Times of Arrival (ToAs) ofSIGINT signals at the plurality of collectors. Estimating the first setof times can occur before the ToAs of the SIGINT signals at theplurality of collectors are received

The technique 1400 can include removing a tracklet 106A-C of theplurality of tracklets so as to not estimate the first set of timesbased on the removed tracklet 106A-C if the tracklet 106A-C is notwithin an expected range of locations. The technique 1400 can includedetermining a plurality of residual errors, one residual error for eachtracklet 106A-C of the plurality of tracklets per set of SIGINT signals.Each residual error can represent a likelihood that the set of SIGINTsignals originated from a respective object 102A-C on the tracklet106A-C. The residual error can be determined based on (1) the first setof times, (2) the second set of times, or (3) the ToAs of the SIGINTsignals of the set of SIGINT signals at the plurality of collectors.

The technique 1400 can include determining the set of SIGINT signalsoriginated from the tracklet 106A-C that corresponds to a lowestresidual error of the plurality of residual errors. The technique 1400can include interpolating where an emitter would have been on a trackletof the plurality of tracklets if a time resolution of the video data isless than a time resolution of SIGINT observations at a collector 108 ofthe plurality of collectors or interpolating a ToA of a signal at thecollector 108 of the plurality collectors if a time resolution of thevideo data is greater than a time resolution of SIGINT signalobservations at the collector 108.

The technique 1400 can include calculating an expected delay based ontwo times of the first set of times, wherein the expected delayindicates how much time is expected to pass between the SIGINT signalbeing received at a first collector 108 of the plurality of collectorsand a second collector 108 of the plurality of collectors, wherein thefirst and second collectors are different collectors. The residual errorcan be determined based on the expected delay and an actual observeddelay determined based on actual ToAs of the SIGINT signal at the firstand second collectors.

The presented techniques, apparatuses, or systems for associating SIGINTsignals 104A-C to tracklets 106A-C of moving emitters can exploitreadily available SIGINT observable data and tracklet 106A-C data. TheSIGINT observable data can originate from various collectors 108.

Simulation results indicate the benefit of transforming tracklet 106A-Cto an arrival time at a collector 108 for a case of simple relativemotion between a set of moving objects 102A-C, multiple collectors 108,and sets of associated SIGINT events. More complex motion, such asstarting and stopping and variation in speed, can be beneficial to theprocess if it results in diversity between the prospective movers (e.g.,moving objects 102A-C).

An association of SIGINT signals 104A-C to tracklet 106A-C or object102A-C can be used in a variety of applications. The residuals generatedfor association hypotheses can be used to support defragmentation oftracks in a multi hypothesis tracker. The association process by itselfcan determine which object 102A-C is most likely associated with aspecific collection of associated SIGINT events, such as to help supportfuture detection of a specific object 102A-C. These approaches canprovide a mechanism to determine if the collection of SIGINT eventsoriginated from a stationary emitter (e.g., a stationary object 102A-C)and can provide a measure of confidence for the stationary assumption.

The benefits of one or more techniques discussed herein can be limitedby the delay of receiving the SIGINT events (e.g., ToAs at thecollectors 108). This can be due to a timeliness requirement for theSIGINT system, communication limitations between systems or items of thesystem, long integration times, or due to a complex SIGINT associationprocess. One or more presented techniques can limit the impact of SIGINTsignal 104A-C observation delays by reducing the computations that occurafter the arrival of this data.

One or more techniques discussed herein can provide a more timelyapproach to exploiting SIGINT, such as in high density urban areas, suchas for the purpose of improving the location associated with SIGINTevents, or such as to improve a tracking process.

FIG. 15 illustrates a block diagram of an example machine 1500 uponwhich any one or more of the techniques (e.g., methodologies) discussedherein may perform. In alternative embodiments, the machine 1500 mayoperate as a standalone device or may be connected (e.g., networked) toother machines. In a networked deployment, the machine 1500 may operatein the capacity of a server machine, a client machine, or both inserver-client network environments. In an example, the machine 1500 mayact as a peer machine in peer-to-peer (P2P) (or other distributed)network environment. The machine 1500 may be a personal computer (PC), atablet PC, a set-top box (STB), a personal digital assistant (PDA), amobile telephone, a web appliance, a network router, switch or bridge,or any machine capable of executing instructions (sequential orotherwise) that specify actions to be taken by that machine, such as abase station. Further, while only a single machine is illustrated, theterm “machine” shall also be taken to include any collection of machinesthat individually or jointly execute a set (or multiple sets) ofinstructions to perform any one or more of the methodologies discussedherein, such as cloud computing, software as a service (SaaS), othercomputer cluster configurations.

Examples, as described herein, may include, or may operate on, logic ora number of components, modules, or mechanisms. Modules are tangibleentities (e.g., hardware) capable of performing specified operationswhen operating. A module includes hardware. In an example, the hardwaremay be specifically configured to carry out a specific operation (e.g.,hardwired). In an example, the hardware may include configurableexecution units (e.g., transistors, circuits, etc.) and a computerreadable medium containing instructions, where the instructionsconfigure the execution units to carry out a specific operation when inoperation. The configuring may occur under the direction of theexecutions units or a loading mechanism. Accordingly, the executionunits are communicatively coupled to the computer readable medium whenthe device is operating. In this example, the execution units may be amember of more than one module. For example, under operation, theexecution units may be configured by a first set of instructions toimplement a first module at one point in time and reconfigured by asecond set of instructions to implement a second module.

Machine (e.g., computer system) 1500 may include a hardware processor1502 (e.g., a central processing unit (CPU), a graphics processing unit(GPU), a hardware processor core, or any combination thereof), a mainmemory 1504 and a static memory 1506, some or all of which maycommunicate with each other via an interlink (e.g., bus) 1508. Themachine 1500 may further include a display unit 1510, an alphanumericinput device 1512 (e.g., a keyboard), and a user interface (UI)navigation device 1514 (e.g., a mouse). In an example, the display unit1510, input device 1512 and UI navigation device 1514 may be a touchscreen display. The machine 1500 may additionally include a storagedevice (e.g., drive unit) 1516, a signal generation device 1518 (e.g., aspeaker), a network interface device 1520, and one or more sensors 1521,such as a global positioning system (GPS) sensor, compass,accelerometer, or other sensor. The machine 1500 may include an outputcontroller 1528, such as a serial (e.g., universal serial bus (USB),parallel, or other wired or wireless (e.g., infrared (IR), near fieldcommunication (NFC), etc.) connection to communicate or control one ormore peripheral devices (e.g., a printer, card reader, etc.).

The storage device 1516 may include a machine readable medium 1522 onwhich is stored one or more sets of data structures or instructions 1524(e.g., software) embodying or utilized by any one or more of thetechniques or functions described herein. The instructions 1524 may alsoreside, completely or at least partially, within the main memory 1504,within static memory 1506, or within the hardware processor 1502 duringexecution thereof by the machine 1500. In an example, one or anycombination of the hardware processor 1502, the main memory 1504, thestatic memory 1506, or the storage device 1516 may constitute machinereadable media.

While the machine readable medium 1522 is illustrated as a singlemedium, the term “machine readable medium” may include a single mediumor multiple media (e.g., a centralized or distributed database, and/orassociated caches and servers) configured to store the one or moreinstructions 1524.

The term “machine readable medium” may include any medium that iscapable of storing, encoding, or carrying instructions for execution bythe machine 1500 and that cause the machine 1500 to perform any one ormore of the techniques of the present disclosure, or that is capable ofstoring, encoding or carrying data structures used by or associated withsuch instructions. Non-limiting machine readable medium examples mayinclude solid-state memories, and optical and magnetic media. In anexample, a massed machine readable medium comprises a machine readablemedium with a plurality of particles having resting mass. Specificexamples of massed machine readable media may include: non-volatilememory, such as semiconductor memory devices (e.g., ElectricallyProgrammable Read-Only Memory (EPROM), Electrically ErasableProgrammable Read-Only Memory (EEPROM)) and flash memory devices;magnetic disks, such as internal hard disks and removable disks;magneto-optical disks; and CD-ROM and DVD-ROM disks.

The instructions 1524 may further be transmitted or received over acommunications network 1526 using a transmission medium via the networkinterface device 1520 utilizing any one of a number of transferprotocols (e.g., frame relay, internet protocol (IP), transmissioncontrol protocol (TCP), user datagram protocol (UDP), hypertext transferprotocol (HTTP), etc.). Example communication networks may include alocal area network (LAN), a wide area network (WAN), a packet datanetwork (e.g., the Internet), mobile telephone networks (e.g., cellularnetworks), Plain Old Telephone (POTS) networks, and wireless datanetworks (e.g., Institute of Electrical and Electronics Engineers (IEEE)802.11 family of standards known as Wi-Fi®, IEEE 802.16 family ofstandards known as WiMax®), IEEE 802.15.4 family of standards,peer-to-peer (P2P) networks, among others. In an example, the networkinterface device 1520 may include one or more physical jacks (e.g.,Ethernet, coaxial, or phone jacks) or one or more antennas to connect tothe communications network 1526. In an example, the network interfacedevice 1520 may include a plurality of antennas to wirelesslycommunicate using at least one of single-input multiple-output (SIMO),multiple-input multiple-output (MIMO), or multiple-input single-output(MISO) techniques. The term “transmission medium” shall be taken toinclude any intangible medium that is capable of storing, encoding orcarrying instructions for execution by the machine 1500, and includesdigital or analog communications signals or other intangible medium tofacilitate communication of such software.

EXAMPLES AND NOTES

The present subject matter may be described by way of several examples.

Example 1 can include or use subject matter (such as an apparatus, amethod, a means for performing acts, or a device readable memoryincluding instructions that, when performed by the device, can cause thedevice to perform acts), such as can include or use estimating Times ofArrival (ToAs) at each of a plurality of collectors of a first signalfrom each of a plurality of moving transmitters, each first signaltransmitted from a transmitter on a tracklet extracted from video dataand received at the plurality of collectors, wherein a location of eachof the plurality of collectors is known, comparing each estimated ToA toa respective actual ToA of a SIGnal INTelligence (SIGINT) signalreceived at each of the collectors, or determining a likelihood that thesignal corresponds to the SIGINT signal to determine whether the SIGINTsignal was transmitted from a transmitter on the corresponding tracklet.

Example 2 can include or use, or can optionally be combined with thesubject matter of Example 1, to include or use generating a boundingarea so as to constrain tracklets to tracklets within the bounding area,wherein the bounding area includes a geographical region in which aSIGINT event is estimated to have originated from as a function of anestimated location and corresponding covariance defining a confidencethat the estimated location is the actual location the SIGINT eventoriginated from, wherein each of the plurality of SIGINT events includesa SIGINT signal and the actual ToA of the SIGINT signal.

Example 3 can include or use, or can optionally be combined with thesubject matter of Example 2, to include or use determining a pluralityof residual errors, one residual error for each tracklet in a pluralityof tracklets per SIGINT event, wherein each residual error represents alikelihood that a SIGINT event originated from a respective object onthe tracklet, wherein the residual error is determined as a function of(1) an interpolated location of the transmitter at a specific time, theinterpolated location determined based on the tracklet data, (2) theactual ToAs at each collector of the SIGINT event, and (3) the locationof each collector.

Example 4 can include or use, or can optionally be combined with thesubject matter of Example 3, to include or use determining the SIGINTsignal originated from an object on the tracklet that corresponds to alowest residual error of the plurality of residual errors.

Example 5 can include or use, or can optionally be combined with thesubject matter of at least one of Examples 1-4, to include or usedetermining if it is more likely that the SIGINT signal originated froma moving transmitter or a stationary transmitter.

Example 6 can include or use, or can optionally be combined with thesubject matter of at least one of Examples 1-5, to include or usewherein estimating the ToA at the plurality of collectors includes (1)estimating a first time, the first time indicating how long it wouldtake the signal to travel from a point on the tracklet to a collector ofthe plurality of collectors, (2) determining a second time, the secondtime indicating the time at which the transmitter was at the point onthe tracklet, or (3) determining an estimated ToA at a collector of theplurality of collectors as a function of the first time and the secondtime.

Example 7 can include or use, or can optionally be combined with thesubject matter of at least one of Examples 2-6, to include or usewherein the plurality of tracklets were each active in the bounding areain a time window, the time window determined as a function of the actualToAs of SIGINT signals at the plurality of collectors.

Example 8 can include or use subject matter (such as an apparatus, amethod, a means for performing acts, or a device readable memoryincluding instructions that, when performed by the device, can cause thedevice to perform acts), such as can include or use (1) estimating afirst set of times, each time of the first set of times indicating howmuch time it would take for a respective SIGnal INTelligence (SIGINT)signal of a set of a plurality of SIGINT signals to travel from a pointon a tracklet of a plurality of tracklets extracted from video data to arespective collector of a plurality of collectors at determinablelocations, (2) estimating a second set of times corresponding to timesat which the video data corresponding to the point on the tracklet wasgathered, or (3) associating the set of SIGINT signals with a trackletof the plurality of tracklets based on the first set of times, thesecond set of times, and a set of Times of Arrival (ToAs) of SIGINTsignals at the plurality of collectors.

Example 9 can include or use or can optionally be combined with thesubject matter of at least one of Examples 1-8 to include or useremoving a tracklet of the plurality of tracklets so as to not estimatethe first set of times based on the removed tracklet if the tracklet isnot within an expected range of locations.

Example 10 can include or use or can optionally be combined with thesubject matter of Examples 8-9 to include or use determining a pluralityof residual errors, one residual error for each tracklet of theplurality of tracklets per set of SIGINT signals, wherein each residualerror represents a likelihood that the set of SIGINT signals originatedfrom a respective object on the tracklet, wherein the residual error isdetermined based on (1) the first set of times, (2) the second set oftimes, or (2) the ToAs of the SIGINT signals of the set of SIGINTsignals at the plurality of collectors.

Example 11 can include or use or can optionally be combined with thesubject matter of Example 10 to include or use determining the set ofSIGINT signals originated from the tracklet that corresponds to a lowestresidual error of the plurality of residual errors.

Example 12 can include or use or can optionally be combined with thesubject matter of Examples 8-11 to include or use interpolating where anemitter would have been on a tracklet of the plurality of tracklets if atime resolution of the video data is less than a time resolution ofSIGINT observations or interpolating a ToA of a signal at a collector ofthe plurality collectors if a time resolution of the video data isgreater than a time resolution of SIGINT signal observations at thecollector.

Example 13 can include or use or can optionally be combined with thesubject matter of Examples 8-12 to include or use calculating anexpected delay based on two times of the first set of times, wherein theexpected delay indicates how much time is expected to pass between theSIGINT signal being received at a first collector of the plurality ofcollectors and a second collector of the plurality of collectors,wherein the first and second collectors are different collectors, andwherein the residual error is determined based on the expected delay andan actual observed delay determined based on actual ToAs of the SIGINTsignal at the first and second collectors.

Example 14 can include or use or can optionally be combined with thesubject matter of Examples 8-13 to include or use wherein estimating thefirst set of times occurs before the ToAs of the SIGINT signal at theplurality of collectors are received.

Example 15 can include or use, or can be optionally be combined with thesubject matter of at least one of Examples 9 or 11-14, to includesubject matter (such as an apparatus, a method, a means for performingacts, or a device readable memory including instructions that, whenperformed by the device, can cause the device to perform acts), such ascan include or use (1) estimating a first set of times, each time of thefirst set of times indicating how much time it would take for arespective SIGnal INTelligence (SIGINT) signal of a set of a pluralityof SIGINT signals to travel from a tracklet of a plurality of trackletsextracted from video data to a respective collector of a plurality ofcollectors at determinable locations, (2) adding each time of the firstset of times to a respective second time to create a set of third times,wherein the respective second time is a time at which the video datacorresponding to the different point on the tracklet was gathered, or(3) associating the set of SIGINT signals with a tracklet of theplurality of tracklets based on the set of third times and a set ofTimes of Arrival (ToAs) of SIGINT signals at the plurality ofcollectors.

Example 16 can include or use or can optionally be combined with thesubject matter of Example 15 to include or use determining a pluralityof residual errors, a residual error for each tracklet of the pluralityof tracklets per set of SIGINT signals, wherein each residual errorrepresents a likelihood that the set of SIGINT signals originated from arespective object on the tracklet, wherein the residual error isdetermined based on (1) the set of third times, or (2) the ToAs of theSIGINT signals of the set of SIGINT signals at the plurality ofcollectors.

Example 17 can include or use, or can be optionally be combined with thesubject matter of at least one of Examples 9 or 11-14, to includesubject matter (such as an apparatus, a method, a means for performingacts, or a device readable memory including instructions that, whenperformed by the device, can cause the device to perform acts), such ascan include or use (1) estimate a first set of times, each time of thefirst set of times indicating how much time it would take for arespective SIGnal INTelligence (SIGINT) signal of a set of a pluralityof SIGINT signals to travel from a respective different point on atracklet of a plurality of tracklets extracted from video data to acollector of a plurality of collectors at determinable locations, or (2)associate the set of SIGINT signals with a tracklet of the plurality oftracklets based on the first set of times and a set of Times of Arrival(ToAs) of SIGINT signals at the plurality of collectors.

Example 18 can include or use or can optionally be combined with thesubject matter of Example 17 to include or use determining a pluralityof residual errors, one residual error for each tracklet of theplurality of tracklets per set of SIGINT signals, wherein each residualerror represents a likelihood that the set of SIGINT signals originatedfrom a respective object on the tracklet, wherein the residual error isdetermined based on (1) the first set of times and (2) the ToAs of theSIGINT signals of the set of SIGINT signals at the plurality ofcollectors.

Example 19 can include or use or can optionally be combined with thesubject matter of at least one of Examples 1-18 to include or use aprocessor configured to perform any one or more of the operation(s) ofany one of Examples 1-18.

The above Description of Embodiments includes references to theaccompanying drawings, which form a part of the detailed description.The drawings show, by way of illustration, specific embodiments in whichmethods, apparatuses, and systems discussed herein may be practiced.These embodiments are also referred to herein as “examples.” Suchexamples may include elements in addition to those shown or described.However, the present inventors also contemplate examples in which onlythose elements shown or described are provided. Moreover, the presentinventors also contemplate examples using any combination or permutationof those elements shown or described (or one or more aspects thereof),either with respect to a particular example (or one or more aspectsthereof), or with respect to other examples (or one or more aspectsthereof) shown or described herein.

The flowchart and block diagrams in the FIGS. illustrate thearchitecture, functionality, and operation of possible implementationsof systems, methods and computer program products according to variousaspects of the present disclosure. In this regard, each block in theflowchart or block diagrams may represent a module, segment, or portionof code, which comprises one or more executable instructions forimplementing the specified logical function(s). It should also be notedthat, in some alternative implementations, the functions noted in theblock may occur out of the order noted in the figures. For example, twoblocks shown in succession may, in fact, be executed substantiallyconcurrently, or the blocks may sometimes be executed in the reverseorder, depending upon the functionality involved. It will also be notedthat each block of the block diagrams and/or flowchart illustration, andcombinations of blocks in the block diagrams and/or flowchartillustration, may be implemented by special purpose hardware-basedsystems that perform the specified functions or acts, or combinations ofspecial purpose hardware and computer instructions.

The functions or techniques described herein may be implemented insoftware or a combination of software and human implemented procedures.The software may consist of computer executable instructions stored oncomputer readable media such as memory or other type of storage devices.The term “computer readable media” is also used to represent any meansby which the computer readable instructions may be received by thecomputer, such as by different forms of wired or wireless transmissions.Further, such functions correspond to modules, which are software,hardware, firmware or any combination thereof. Multiple functions may beperformed in one or more modules as desired, and the embodimentsdescribed are merely examples. The software may be executed on a digitalsignal processor, ASIC, microprocessor, or other type of processoroperating on a computer system, such as a personal computer, server orother computer system.

In this document, the terms “a” or “an” are used, as is common in patentdocuments, to include one or more than one, independent of any otherinstances or usages of “at least one” or “one or more.” In thisdocument, the term “or” is used to refer to a nonexclusive or, such that“A or B” includes “A but not B,” “B but not A,” and “A and B,” unlessotherwise indicated. In this document, the terms “including” and “inwhich” are used as the plain-English equivalents of the respective terms“comprising” and “wherein.” Also, in the following claims, the terms“including” and “comprising” are open-ended, that is, a system, device,article, composition, formulation, or process that includes elements inaddition to those listed after such a term in a claim are still deemedto fall within the scope of that claim. Moreover, in the followingclaims, the terms “first,” “second,” and “third,” etc. are used merelyas labels, and are not intended to impose numerical requirements ontheir objects.

As used herein, a “-” (dash) used when referring to a reference numbermeans “or”, in the non-exclusive sense discussed in the previousparagraph, of all elements within the range indicated by the dash. Forexample, 103A-B means a nonexclusive “or” of the elements in the range{103A, 103B}, such that 103A-103B includes “103A but not 103B”, “103Bbut not 103A”, and “103A and 103B”.

The above description is intended to be illustrative, and notrestrictive. For example, the above-described examples (or one or moreaspects thereof) may be used in combination with each other. Otherembodiments may be used, such as by one of ordinary skill in the artupon reviewing the above description. The Abstract is provided to complywith 37 C.F.R. §1.72(b), to allow the reader to quickly ascertain thenature of the technical disclosure. It is submitted with theunderstanding that it will not be used to interpret or limit the scopeor meaning of the claims. Also, in the above Description of Embodiments,various features may be grouped together to streamline the disclosure.This should not be interpreted as intending that an unclaimed disclosedfeature is essential to any claim. Rather, inventive subject matter maylie in less than all features of a particular disclosed embodiment.Thus, the following claims are hereby incorporated into the Descriptionof Embodiments as examples or embodiments, with each claim standing onits own as a separate embodiment, and it is contemplated that suchembodiments may be combined with each other in various combinations orpermutations. The scope of the invention should be determined withreference to the appended claims, along with the full scope ofequivalents to which such claims are entitled.

What is claimed is:
 1. A method for associating tracklets with SIGnal INTelligence (SIGINT) signals, the method comprising: receiving, at a processing unit, the SIGINT signals and the tracklets, a tracklet for each moving object of a plurality of objects in video data, the SIGINT signals including respective actual Time of Arrivals (ToAs) indicating a time at which the SIGINT signal arrived at a collector of a plurality of collectors at determinable locations; estimating, using the processing unit, a first set of times, each time of the first set of times indicating how much time it takes for a respective SIGnal INTelligence (SIGINT) signal of SIGINT signals to travel from a point on a tracklet of the tracklets to a respective collector of the plurality of collectors; estimating a second set of times, times in the second set of times corresponding to times at which the video data corresponding to respective points on the tracklet were gathered; determining a residual error associated with each tracklet by comparing the actual ToAs to a third set of times determined by adding the first set of times to the second of times; and associating the SIGINT signals with a respective tracklet of the tracklets associated with the smallest residual error.
 2. The method of claim 1, further comprising: removing a tracklet of the plurality of tracklets so as to not estimate the first set of times based on the removed tracklet if the tracklet is not within an expected range of locations.
 3. The method of claim 2, further comprising: determining a plurality of residual errors, one residual error for each tracklet of the plurality of tracklets per set of SIGINT signals, wherein each residual error represents a likelihood that the set of SIGINT signals originated from a respective object on the tracklet, wherein the residual error is determined based on (1) the first set of times, (2) the second set of times, and (2) the ToAs of the SIGINT signals of the set of SIGINT signals at the plurality of collectors.
 4. The method of claim 3, further comprising: determining the set of SIGINT signals originated from the tracklet that corresponds to a lowest residual error of the plurality of residual errors.
 5. The method of claim 4, further comprising: interpolating where an emitter would have been on a tracklet of the plurality of tracklets if a time resolution of the video data is less than a time resolution of SIGINT observations or interpolating a ToA of a signal at a collector of the plurality collectors if a time resolution of the video data is greater than a time resolution of SIGINT signal observations at the collector.
 6. The method of claim 5, further comprising: calculating an expected delay based on two times of the first set of times, wherein the expected delay indicates how much time is expected to pass between the SIGINT signal being received at a first collector of the plurality of collectors and a second collector of the plurality of collectors, wherein the first and second collectors are different collectors, and wherein the residual error is determined based on the expected delay and an actual observed delay determined based on actual ToAs of the SIGINT signal at the first and second collectors.
 7. The method of claim 6, wherein estimating the first set of times occurs before the ToAs of the SIGINT signal at the plurality of collectors are received.
 8. A non-transitory computer readable storage device including instructions stored thereon, the instructions, which when executed by a machine, cause the machine to perform operations for associating tracklets with SIGnal INTelligence (SIGINT) signals, the operations comprising: receiving the SIGINT signals and the tracklets, a tracklet for each moving object of a plurality of objects in video data, the SIGINT signals including respective actual Time of Arrivals (ToAs) indicating a time at which the SIGINT signal arrived at a collector of a plurality of collectors at determinable locations; estimating a first set of times, each time of the first set of times indicating how much time it takes for a respective SIGnal INTelligence (SIGINT) signal of the SIGINT signals to travel from a tracklet of the tracklets to a respective collector of the plurality of collectors; adding each time of the first set of times to a respective second time in a second set of times to create a set of third times, wherein the respective second time is a time at which the video data corresponding to the different point on the tracklet was gathered; determining a residual error associated with each tracklet by comparing the actual ToAs to a third set of times determined by adding the first set of times to the second of times; and associating the SIGINT signals with a respective tracklet of the tracklets associated with the smallest residual error.
 9. The storage device of claim 8, further comprising instructions, which when executed by the machine, cause the machine to perform operations comprising: removing a tracklet of the plurality of tracklets so as to not estimate the first set of times based on the removed tracklet if the tracklet is not within an expected range of locations.
 10. The storage device of claim 9, further comprising instructions, which when executed by the machine, cause the machine to perform operations comprising: determining a plurality of residual errors, a residual error for each tracklet of the plurality of tracklets per set of SIGINT signals, wherein each residual error represents a likelihood that the set of SIGINT signals originated from a respective object on the tracklet, wherein the residual error is determined based on (1) the set of third times, and (2) the ToAs of the SIGINT signals of the set of SIGINT signals at the plurality of collectors.
 11. The storage device of claim 10, further comprising instructions, which when executed by the machine, cause the machine to perform operations comprising: determining the set of SIGINT signals originated from the tracklet that corresponds to a lowest residual error of the plurality of residual errors.
 12. The storage device of claim 11, further comprising instructions, which when executed by the machine, cause the machine to perform operations comprising: interpolating where an emitter would have been on a tracklet of the plurality of tracklets if a time resolution of the video data is less than a time resolution of SIGINT observations or interpolating a ToA of a hypothetical signal at a collector of the plurality collectors if a time resolution of the video data is greater than a time resolution of SIGINT observations at the collector.
 13. The storage device of claim 12, further comprising instructions, which when executed by the machine, cause the machine to perform operations comprising calculating an expected delay based on two times of the first set of times, wherein the expected delay indicates how much time is expected to pass between the SIGINT signal being received at a first collector of the plurality of collectors and a second collector of the plurality of collectors, wherein the first and second collectors are different collectors, and wherein the residual error is determined based on the expected delay and an actual observed delay determined based on actual ToAs of the SIGINT signal at the first and second collectors.
 14. The storage device of claim 13, wherein the instructions for determining the first set of times comprise instructions for determining the first set of times before the ToAs of the SIGINT signal at the plurality of collectors are received.
 15. A system configured to associate tracklets with SIGnal INTelligence (SIGINT) signals, the device comprising: a processor configured to: receive the SIGINT signals and the tracklets, a tracklet for each moving object of a plurality of objects in video data, the SIGINT signals including respective actual Time of Arrivals (ToAs) indicating a time at which the SIGINT signal arrived at a collector of a plurality of collectors at determinable locations; estimate a first set of times, each time of the first set of times indicating how much time it takes for a respective SIGnal INTelligence (SIGINT) signal of the SIGINT signals to travel from a respective different point on a tracklet of the tracklets to a collector of the plurality of collectors; add each time of the first set of times to a respective second time in a second set of times to create a set of third times, wherein the respective second time is a time at which the video data corresponding to the different point on the tracklet was gathered; determine a residual error associated with each tracklet by comparing the actual ToAs to a third set of times determined by adding the first set of times to the second of times; and associate the SIGINT signals with a respective tracklet of the tracklets associated with the smallest residual error.
 16. The device of claim 15, wherein the processor is further configured to: remove a tracklet of the plurality of tracklets so as to not estimate the first set of times based on the removed tracklet if the tracklet is not within an expected range of locations.
 17. The device of claim 16, wherein the processor is further configured to: determine a plurality of residual errors, one residual error for each tracklet of the plurality of tracklets per set of SIGINT signals, wherein each residual error represents a likelihood that the set of SIGINT signals originated from a respective object on the tracklet, wherein the residual error is determined based on (1) the first set of times and (2) the ToAs of the SIGINT signals of the set of SIGINT signals at the plurality of collectors.
 18. The device of claim 17, wherein the processor is further configured to: determine the set of SIGINT signals originated from the tracklet that corresponds to a lowest residual error of the plurality of residual errors.
 19. The device of claim 18, wherein the processor is further configured to: interpolate where an emitter would have been on a tracklet of the plurality of tracklets if a time resolution of the video data is less than a time resolution of SIGINT observations or interpolate a ToA of a hypothetical signal at a collector of the plurality collectors if a time resolution of the video data is greater than a time resolution of SIGINT observations at the collector.
 20. The device of claim 19, wherein the processor is further configured to: calculate an expected delay based on two times of the first set of times, wherein the expected delay indicates how much time is expected to pass between the SIGINT signal being received at a first collector of the plurality of collectors and a second collector of the plurality of collectors, wherein the first and second collectors are different collectors, and wherein the residual error is determined based on the expected delay and an actual observed delay determined based on actual ToAs of the SIGINT signal at the first and second collectors. 